Lecture 9 — turning the resiliency patterns from last class into a repeatable operational loop that catches failure, reacts to it, and gets a little smarter every time.
"Hope is not a strategy — but neither is a circuit breaker nobody is watching." — why resiliency needs a process, not just patterns
Redundancy and failover only help if a human — or an automated system — notices the failure, reacts to it, and learns from it. That's exactly what the resilience process gives us.
The resilience process is the operational habit that makes redundancy, failover and auto-scaling actually work in practice — a loop a team runs every time something goes wrong, so the system gets steadily harder to break.
Continuous monitoring of metrics, logs, and traces watches for signs that something has drifted from normal — latency spikes, error rates, failed health checks.
A detected anomaly is turned into an alert: it's routed, prioritised, and pushed to whoever is on-call — through chat, paging apps, or phone calls for the most severe cases.
The on-call engineer (or an automated system) acts to restore service — a rollback, a failover, a restart, or a scripted remediation — guided by a runbook where one exists.
Once service is restored, the team runs a blameless postmortem: what failed, why detection or response was slow, and what concrete change prevents a repeat.
Recall Lecture 7: Kaizen is continuous, incremental improvement made by the people doing the work. Refine is that habit applied directly to reliability — every incident makes the system a little harder to break next time.
| Stage | Category | Example tools |
|---|---|---|
| Detect | Metrics, logs & tracing | Prometheus, Grafana, ELK Stack, Datadog |
| Alert | Paging & notification | PagerDuty, Opsgenie, Slack/Teams integrations |
| Respond | Runbooks & automation | Ansible, scripted rollback, ChatOps bots |
| Refine | Postmortem & tracking | Shared incident docs, Jira/GitLab Tracker action items |
Tool names matter less than the discipline: each stage needs an owner, even if the "tool" is a shared document.
How long from failure starting to it being noticed.
How long from alert firing to a human acknowledging it.
How long from failure to service being fully restored.
How long, on average, the system runs before the next failure.
A healthy resilience process drives MTTD, MTTA and MTTR down over time — and pushes MTBF up. That trend is the real evidence that "Refine" is working.
A microservice from Lecture 8's "Orders" service starts timing out under heavy load. Walk through all four stages: what would Detect catch first, who should Alert reach, what's a reasonable first Response, and what's one concrete Refine action item for next sprint?
Refine is Kaizen applied directly to incidents — small, continuous fixes after every failure.
Redundancy, circuit breakers and failover are the raw material the Respond stage actually uses.
The operational loop that makes all of the above real, every single day, on call.
DevOps principles; Version Control (SVN, Git, GitHub); Gitflow workflow; CI with GitHub Actions.
Think of a time something broke and nobody noticed right away — which stage of this loop was missing?